Providing business intelligence over huge data volumes is an overwhelming challenge. Organizations generate and collect terabytes of transactions, log records, event data, traffic information, textual messages, transcribed conversations and more mission critical information every day, but struggle when attempting to analyze, search and gain intelligence based on the data. The task is becoming more and more challenging as the typical data warehouse usage model is changing. With rapid growth in the volumes of aggregated data, increasing number of users, new types of queries and a growing need for real time or near real time analytics, the legacy relational database management systems (RDMBS) based data warehouse solutions are becoming more and more inadequate for the changing needs in the online analytical processing (OLAP) domain.
The scenarios of organizations dealing with billions of events generated or collected daily are becoming more and more common.
The legacy technology underpinning the majority of the data warehouse market carries many deficiencies such as high time and resources consuming load processes, high analytic latency, long queries execution time, limited analytic capabilities, non-linear scalability, growing hardware costs and growing maintenance costs.
Bitmaps variants are typically used to implement inverted indexes ids lists when there is a need to maintain and operate over very large and usually dense ids sets. Hierarchical bitmaps and moreover compressed hierarchical bitmaps are less commonly used, partly due to the complexity of the implementation, and while CHB enables to operate efficiently over spare sets as well as dense sets it is considered wasteful for small sets. Its main advantage is efficient union and intersection operations over large sets. In addition to complexity of implementation hierarchical bitmaps were also considered less optimized for disk reads and writes due to the fact that the entire data structure must be loaded into memory for retrieval and dump into the disk entirely for each update.
This invention brings the SHB structure that revolutionize hierarchical bitmaps from a data structure for lists intersection and union into full index for sets of any size (even huge ones) that enable O(1) search time with extremely low memory requirements compare to legacy index structure such as B-Tree, binary search tree and hash. SHB is a most efficient indexing method that may be used, but not limited to, as a full index for database and generally as quick access index to elements in huge data sets.